Exploratory Study of Lexical Patterns in Multimodal Cues

Multimodal cues present in human-human dialogue help us to interpret other people’s utterances. We undertake an exploratory study into the relationships of multimodal cues and communicative acts, i.e. between what people say and what people do when interacting with one another. If any such patterns are found and can be recognised, they could be exploited to support the understanding of multimodal interaction behaviours and interface design. Initial analysis of lexical categories and hand/arm gestures suggests some categories are more strongly associated with certain gesture types, in particular nouns and pronouns are emphasised in 87% of multimodal production acts involving deictic gestures.